Efficient Inference in Markov Control Problems

Markov control algorithms that perform smooth, non-greedy updates of the policy have been shown to be very general and versatile, with policy gradient and Expectation Maximisation algorithms being particularly popular. For these algorithms, marginal inference of the reward weighted trajectory distribution is required to perform policy updates. We discuss a new exact inference algorithm for these marginals in the finite horizon case that is more efficient than the standard approach based on classical forwardbackward recursions. We also provide a principled extension to infinite horizon Markov Decision Problems that explicitly accounts for an infinite horizon. This extension provides a novel algorithm for both policy gradients and Expectation Maximisation in infinite horizon problems. 1 MARKOV DECISION PROBLEMS A Markov Decision Problem (MDP) is described by an initial state distribution p1(s1), transition distributions p(st+1|st, at) and reward function Rt(st, at), where the state and action at time t are denoted by st and at respectively (Sutton and Barto, 1998). The state and action spaces can be either discrete or continuous. For a discount factor γ the reward is defined as Rt(st, at) = γR(st, at) for a stationary reward R(st, at), where γ ∈ [0, 1). We assume a stationary policy, π, defined as a set of conditional distributions over the action space, πa,s = p(at = a|st = s, π). The total expected reward of the MDP (the policy utility) To avoid cumbersome notation we also use the notation zt = {st, at} to denote a state-action pair. We use the bold typeface, zt, to denote a vector. is given by